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Use  of  a  Photosimulation  Laboratory  for  Estimating  Vehicle  Detection  Probability 

and  Comparing  Detection  Metrics  (U) 


Thomas  J.  Meitzler*,  Ph.D.,  David  Bednarz,  Ph.D.,  Darryl  Bryk,  Kimberly  Lane,  and  Euijung  Sohn 

Survivability  Technology  Area 

U.S.  Army  Tank-automotive  and  Armaments  Command  (TACOM) 

Warren,  MI,  48397-5000 


Abstract  (U) 

(U)  A  method  is  described  for  using  a  photosimulation  laboratory  environment  to  compare  detection 
metrics  and  evaluate  the  effectiveness  of  camouflage  for  military  vehicles.  There  are  distinct  advantages 
to  acquiring  images  at  the  field  site  and  then  bringing  them  back  to  a  laboratory  environment  for 
observer  testing  versus  taking  the  subjects  out  to  the  field  for  estimating  detection  probability. 
Laboratory  testing  using  field  acquired  imagery  provides  a  repeatable,  secure,  and  relatively  low-cost 
way  to  generate  consistent  data  for  the  measurement  of  the  effectiveness  of  camouflage  relative  to  a 
baseline  vehicle,  and  the  calibration  and  validation  of  target  acquisition  models.  A  laboratory  test 
procedure  is  described  by  the  authors  in  which  a  baseline  Light  Armored  Vehicle  (LAV)  is  compared  to 
a  treated  LAV  in  the  TACOM  Visual  Perception  Laboratory  (VPL)  using  imagery  collected  from  the 
field  in  the  manner  prescribed  by  an  experimental  design. 


1  Introduction  (U)  The  purpose  of  this  paper  is  to  describe  a  methodology  for 
photosimulation  tests  in  a  laboratory  environment,  for  the  purpose  of  vehicle  treatment  comparison  or 
detection  metric  trade-off  studies,  using  imagery  of  the  vehicle  taken  at  the  field  site.  Mega-Pixel,  high- 
resolution,  digital  cameras  presently  available  on  the  market  have  come  very  close  to  equaling  the 
resolution  and  color  depth  attainable  with  film.  Six  megabyte  CCD  imaging  chips  along  with  the  ability 
to  capture  imagery  in  raw  24-bit  format,  combined  with  high  capacity,  portable,  storage  devices  enable 
high-resolution  imagery  to  be  captured  at  field  site  locations  and  easily  delivered  back  to  the  laboratory. 
The  time  consuming  processing  loop  required  with  film  has  been  removed.  Using  high-resolution 
graphics  projectors,  the  imagery  can  then  be  presented  in  the  controlled  environment  of  a  laboratory  in 
such  a  manner  as  to  obtain  observer  data  with  confidence  levels  approaching  99%.  The  benefits 
achieved  using  the  repeatability  and  randomization  offered  by  the  lab  environment  are  not  available  in  a 
traditional  field  test.  The  laboratory  randomization  of  the  order  of  the  stimuli  removes  any  potential  bias 
introduced  by  the  order  of  the  presentation  of  the  stimuli  in  a  traditional  field  test  where  this  type  of 
randomization  is  not  practical. 

(U)  The  test  procedure  described  in  this  report  was  undertaken  as  part  of  the  LAV  Service  Life 
Extension  Program  (SLEP).  Specifically,  the  purpose  of  this  experiment  was  to  determine  the 
performance  of  a  camouflage  treatment  in  reducing  the  probability  of  detection  in  the  visual  part  of  the 
electromagnetic  spectrum  at  various  ranges,  aspect  angles  and  lighting  conditions.  Only  the  unclassified 
baseline  results  will  be  described  in  this  report. 

2  Experiment  (U) 

(U)  A  full-factorial  test  matrix  was  developed  and  24-bit  color  imagery  of  the  vehicles  was  collected 
at  the  field  site  using  a  Kodak  460  digital  camera.  The  images  taken  at  the  field  site  were 
prepared  for  the  laboratory  photosimulation  test  and  then  presented  to  thirty  subjects.  The 
experimental  factors  and  levels  with  their  values  are  shown  below  in  Table  1.  The 
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photosimulation  test  in  the  lab  was  arranged  so  that  the  pixel  Instantaneous  Field-Of-View 
(IFOV)  subtended  by  the  monitors  was  less  than  one  minute  of  arc  and  the  displayed  image 
represented  a  unity  magnification  or  IX  representation  to  the  subject.  The  first  test  was  meant  to 
emulate  naked-eye  vision.  Prior  to  the  actual  test,  the  subjects  were  instructed  on  the  purpose  of 
the  test  and  given  a  pre-test  in  which  they  could  become  familiar  with  the  imagery  and  software. 
None  of  the  pictures  used  in  the  pretest  were  used  in  the  actual  test,  however,  the  images  were 
from  the  same  set.  The  test  protocol  was  to  display  an  image  with  a  time-out  of  thirty  seconds. 

The  imagery  was  cropped  so  that  no  scrolling  was  required.  The  target  can  appear  within  one  of 
five  possible  regions.  The  soldier  must  use  the  mouse  to  “click-on”  what  he  or  she  thinks  is  a 
target,  based  on  the  training.  The  tests  are  done  in  a  dark  room  in  which  the  subjects  are  ‘dark- 
adapted’  to  maximize  contrast  differences  in  the  images. 

(U)  Analysis  of  the  first  test  showed  most  subjects  obtained  a  score  of  only  20  %  detection.  This  is 
not  unreasonable  given  the  difficulty  of  the  imagery.  The  ranges  are  not  unusual  for  such  a  test, 
however  the  high  degree  of  clutter,  and  in  particular,  the  height  of  the  grass  on  the  terrain  makes 
it  difficult  for  the  unaided  eye  to  detect  common  cue  features  of  the  vehicle.  A  second  test  was 
arranged  to  simulate  closer  ranges.  The  ability  to  resample  the  imagery  is  a  feature  of  lab  testing 
that  would  not  be  available  on  the  field.  Another  benefit  of  the  laboratory  environment  is  that 
atmospheric  effects  can  be  added  to  the  imagery  for  a  more  controlled  simulation  of  atmospheric 
effects.  Additional  atmospheric  effects  were  not  added  in  this  particular  experiment,  however, 
the  capability  does  exit.  The  imagery  from  the  field  was  of  sufficient  resolution  so  that  there  was 
no  noticeable  increase  in  pixelation  of  the  imagery.  The  software  used  linear  interpolation  to 
zoom  in  on  the  selected  imagery.  The  presentation  in  the  lab  was  randomized  for  each  subject. 

(U)  At  this  point,  it’s  instructive  to  digress  for  a  moment  to  emphasize  the  rationale  for  using  the 
type  of  design  methodology  described  by  the  authors  in  this  paper.  Statistically  based,  experimental 
design  is  a  strategy  of  designing  experiments  in  such  a  manner  as  to  develop  a  robust  test  plan.  In  other 
words,  a  test  plan  that  is  minimally  affected  by  external  sources  of  variability.  What  makes  vehicle 
detection  experiments  so  challenging  is  that  there  are  many  variables  that  are  present  and  must  be 
accounted  for.  In  addition,  the  variables  interact  with  each  other.  The  correct  approach  to  working  with 
several  factors  is  to  conduct  a  factorial  experiment.  A  factorial  experiment  is  an  experimental  strategy 
in  which  the  factors  are  varied  together,  rather  than  one  at  a  time.  The  factorial  experimental  design 
concept  is  extremely  important  and  powerful  when  used  correctly.  There  are  many  classical  and 
standard  books  on  experimental  design  such  as  Analysis  of  Variance'1  by  Scheffe  and  Experimental 
Designs  2  by  Cochran  and  Cox.  A  contemporary  textbook  by  Montgomery,  Design  and  Analysis  of 
Experiments  3  has  many  useful  examples  contained  in  it.  For  these  type  of  vehicle  tests,  it  is  unwise  to 
leave  out  points  in  the  data  matrix  in  the  hope  of  making  the  test  plan  more  expedient  unless  one  is 
doing  a  well  designed  fractional  factorial. 

(U)When  making  inferences  about  differences  in  a  factor  in  a  perception  experiment  in  the 
laboratory  we  want  to  make  the  experimental  error  as  small  possible.  This  requires  that  we  remove  the 
variability  between  subjects  from  the  experimental  error.  The  design  we  use  to  accomplish  this  is  a 
factorial  experiment  run  in  a  randomized  complete  block.  By  using  this  design  with  the  subjects  as 
blocks  we  form  a  more  homogeneous  experimental  unit  on  which  to  compare  different  factors.  This 
experimental  design  improves  the  accuracy  of  the  comparisons  among  the  different  factors  by 
eliminating  the  variability  among  the  subjects.  Within  a  block,  the  order  in  which  the  treatment 
combinations  are  run  is  randomly  determined.  In  other  words,  for  each  subject,  the  order  of  the 
presentation  if  the  imagery  is  different.  It  is  usually  not  practical  to  implement  this  experimental  design 
in  the  context  of  a  traditional  field  test. 


(U)Table  1  below  shows  the  factors  that  were  decided  to  be  the  most  important  for  the  test  on  the 
vehicle  detection  in  the  field.  The  chosen  factors  are;  1)  Region  of  the  field-of-view  (FOV)  in  which  the 
vehicle  is  present,  2)  the  range  from  the  vehicle  to  the  sensor,  3)  the  type  of  vehicle,  4)  the  aspect  angle 
relative  to  the  observer,  5)  the  lighting  condition  indicated  by  the  position  of  the  sun,  front  lit  or  back  lit, 
and  finally,  6)  the  general  condition  of  the  sky,  clear  or  cloudy.  In  the  vehicle  type  category,  SLEP 
means  the  Service  Life  Extension  Program,  ADCAM  is  the  trade  name  of  the  camouflage. 


(U) 


Region 

1 

Top -Left 

2 

Top-Right 

3 

Lower-Left 

4 

Lower-Right 

5 

Center 

Range  (km) 

1 

1 

2 

1.5 

3 

2 

4 

2.5 

5 

3 

Aspect  angle 

1 

Front 

2 

30  degree 

3 

Side 

Vehicle  Type 

1 

Baseline  (old  LAV) 

2 

SLEP  +  ADCAM 

3 

SLEP  +  ADCAM  -  ADCAM 

bowplane 

Weather  condition 

1 

Clear 

2 

Overcast 

Lighting 

1 

Front  Lit 

2 

Back  Lit 

Table  1:  Factor  matrices  for  the  visual  detection  test  (U) 

(U)The  pictures  below  in  Fig.  1  and  Fig.  2  were  used  for  training  observers  as  to  what  kind  of 
vehicles  they  would  be  looking  for  in  the  test  and  also  indicate  some  of  the  variables  in  the  experiment 
such  as  vehicle  type,  treated  or  untreated,  and  aspect  angle. 


Fig.  1:  Sample  baseline  LAV  training  images  (U) 


Fig.  2:  Treated  LAV  training  images  (U) 

(U)The  figure  below  is  of  the  background  at  the  field  site  and  does  not  have  a  vehicle  in  it.  The 
picture  shows  that  the  grass  height  was  high  at  the  test  site.  The  range  of  the  test  field  was  about  3.5  km 
and  that  the  grass  was  high  and  obstructed  the  view  of  the  vehicle  at  large  ranges,  thus  requiring  the 
simulation  of  powered  optics  by  magnifying  the  images  at  certain  amount  depending  on  initial  range. 


Fig.  3:  Background  image  of  test  site  (U) 

(U)The  charts  in  Fig.  4  below  show  the  X  and  Y  chromaticity  values  of  the  monitors  that  were 
used  in  the  photosimulation  test.  The  color  chromaticity  values  were  measured  in  the  lab  using  a 
spectrophotometer.  The  values  measured  as  projected  on  the  monitors  were  compared  to  photometric 
standard  values  and  found  to  be  virtually  identical.  Based  on  the  similarity  of  photometric  measurements 
between  standards  in  the  field  and  displayed  on  two  identical  SONY  the  monitors,  the  authors  are 
confident  that  the  color  fidelity  is  accurate.  The  primary  physical  difference  of  field  versus  lab  tests  is 
the  level  of  luminance  in  the  lab  as  compared  to  the  field  setting.  The  use  of  low  light  cancels  out  the 
effects  experienced  in  the  field  such  as  glare  and  pupil  size.6  Typically  when  detection  tests  are  done  in 
the  laboratory  environment,  the  subjects  are  dark  adapted  and  the  tests  are  run  under  very  dim  lighting. 
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Figure  4  :  Laboratory  versus  field  chromaticities  (U) 


Analysis  (U) 
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(U)  Below  in  Table  2  is  the  ANOVA  table  for  the  baseline  vehicle  and  the  other  experimental 
factors.  The  analysis  for  the  treated  vehicle  has  been  excluded  because  of  security  classification.  The 
power  of  the  experimental  design  methodology  is  shown  here  in  that  the  significance  of  individual 
factors  and  of  their  interactions  are  available  for  review.  Using  this  kind  of  a  test,  one  can  obtain  not 
only  a  math  model  of  the  detection  probability  versus  any  factor  in  the  test,  but,  one  can  also  obtain  the 
relative  importance  of  the  individual  factors  and  their  interactions  at  various  powers. 

(U)  In  Table  2,  the  first  column  of  the  table  labeled  ‘source’,  is  the  effect  or  factor(s)  in  the 
model,  (only  first  order  interactions  were  considered).  The  second  column  shows  the  type  IV  sum  of 
squares.  Type  VI  Sum  of  Squares  are  used  because  there  are  missing  cells  in  our  design  matrix.  The 
third  column,  labeled  ‘df ,  shows  the  degrees  of  freedom  for  each  sum  of  squares.  The  fourth  column 
labeled  ‘Mean  Square’,  shows  the  mean  square  of  each  effect.  This  is  obtained  by  dividing  the  sum  of 
squares  for  each  effect  by  the  degrees  of  freedom  for  each  effect.  The  fifth  column  is  the  F  statistic  and 
shows  the  F  statistic  for  each  effect.  The  F  statistic  is  obtained  by  dividing  the  mean  square  for  each 
effect  by  the  mean  square  error  term  listed  at  the  bottom  of  the  Mean  Square  column.  Column  six, 
labeled  ‘Sig’,  is  the  P-value  of  the  F  statistic  for  each  effect.  The  smaller  the  P-value  the  greater  the 
importance  of  the  effect.  Table  2  shows  that  the  aspect  angle  was  the  least  important  factor  in  the 
experiment  and  that  subject,  range,  sky  condition,  and  the  interaction  of  range  and  aspect  angle. 

Tests  of  Between-Subjects  Effects 


Dependent  Variable:  RANK  of  RESPONSE 


Source 

Type  IV  Sum 
of  Squares 

df 

Mean  Square 

F 

Siq. 

Corrected  Model 

171 6821 05a 

79 

2173191.202 

17.177 

.000 

Intercept 

1277408367 

1 

1277408367 

10096.792 

.000 

SUBJECT 

26976158.0 

26 

1037544.538 

8.201 

.000 

RANGE 

116354292 

9 

12928254.62 

102.187 

.000 

ASPECT 

1125275.323 

2 

562637.662 

4.447 

.012 

SKY_COND 

2522624.471 

2 

1261312.236 

9.970 

.000 

RANGE  *  ASPECT 

5986618.857 

18 

332589.936 

2.629 

.000 

RANGE  *  SKY_COND 

5272645.363 

18 

292924.742 

2.315 

.001 

ASPECT  *  SKY_COND 

2248729.330 

4 

562182.333 

4.444 

.001 

Error 

223301195 

1765 

126516.258 

Total 

1966792305 

1845 

Corrected  Total 

394983300 

1844 

a-  R  Squared  =  .435  (Adjusted  R  Squared  =  .409) 


Table  2:  ANOVA  of  test  factors  (U) 


(U)  Figure  5  shows  the  model  generated  logistic  curve  of  the  probability  of  detection  versus  the  baseline 
LAV.  This  curve  has  the  effects  of  all  the  various  factors  ‘rolled-up’  into  it.  A  logistic  curve  is  the 
standard  psychometric  function  used  to  model  detection  data. 8 


LAV  25 


Fig.  5:  Logistic  fit  to  the  laboratory  results  (U) 

(U)  The  equation  of  this  curve  fit  is  shown  in  equation  (1)  below, 

predicted  probability  of  detection  =  T - - - T. 

[l +  *-<'•♦**>]  (1) 

(U)  In  equation  (1),  x  is  the  range  from  the  imaging  sensor  to  the  vehicle,  and  the  two  constants 
are  2.154  and  -2.545  respectively.  Fig.  6  is  the  Normal  Q-Q  plot  of  deviance  and  the  graph  shows  the 
plot  of  the  quantiles  of  the  deviances  from  the  logistic  regression  model  against  the  quantiles  of  the 
normal  distribution.  Probability  plots  are  used  to  determine  whether  the  distribution  of  a  variable 
matches  a  given  distribution.  The  points  cluster  around  a  straight  line  if  the  variable  matches  the  given 
distribution. 


Normal  Q-Q  Plot  of  Deviance 


- 1 - 1 - 1 - 1 - 1 - . - 1 
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Observed  Value 


Fig.  6  (U) 

(U)  In  this  analysis,  cases  where  the  subject  did  not  respond  were  not  considered.  A  rank 
transformation  was  applied  to  the  dependent  variable  Response.  This  was  done  to  change  our  scale  of 
measurement  from  ordinal  to  interval/ratio  for  ANOVA  analysis.  The  linear  model  for  this  experiment 
is: 


yLJkl  -fl+Ti  +  3j  +  Pk  +  (r^)y  +  (Tfi)ik+  (dp) jk  +Yl+  £ijkl  (2) 

where, 

T;  represents  the  range  effect, 

Sj  represents  the  aspect  angle  effect, 

(3k  represents  the  sky  condition  effect, 

(T S)ij  represents  the  interaction  effect  of  range  and  aspect  angle, 

(r/3)ik  represents  the  interaction  effect  of  range  and  sky  conditions, 

0 SP)jk  represents  the  interaction  effect  of  aspect  angles  and  sky  conditions, 
yl  represents  the  block  effect  due  to  the  subjects,  and 
£ijkl  is  a  normally  distributed  error  term. 

(U)  The  complete  analysis  of  variance  for  this  experiment  is  summarized  in  Table  2.  All  the 
main  effects  except  for  the  aspect  angle  are  significant  at  the  one  percent  level.  The  interaction  terms 
are  all  significant  at  the  one  percent  level. 


4  Metric  Evaluation  of  the  imagery  (U) 

(U)  The  metric  evaluated  by  the  authors  on  this  data  set  were  image  entropy,  defined  by  equation 
three  below.  Fig.’s  7  and  8  show  how  the  entropy  metric  correlated  to  range  and  Pd. 


#= -£/>(£)  log2P(#) 

S=  0 

where  p(g)isthe  probability  of  grey  scale  value  g ,  and  the  range  of  (3) 

gis[0...,L-l]. 


Fig.  7:  Image  entropy  versus  target  to  sensor  range  (U) 


Entropy 

Fig.  8:  Pd  versus  Entropy 
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5  Conclusions  (U) 

(U)  In  summary,  an  advantage  of  using  the  photosimulation  lab  environment  is  that 
experimenters  are  able  to  archive  scenes  used  in  the  simulation,  so  that,  at  a  later  time  it  is  possible  to 
rerun  the  same  image  data  set  on  a  different  subject  pool.  The  new  subjects  may  have  different  training 
and  the  images  may  also  be  modified  by  either  magnification  or  adding  atmospheric  conditions.  This 
provides  tremendous  cost  savings  since  there  is  no  need  to  pay  for  another  field  test. 

(U)  An  experimental  design  method  was  used  to  design  an  imagery  collection  test  plan  and  a 
laboratory  photosimulation  testing  procedure.  The  probability  of  detection  was  determined  for  the 
baseline  and  treated  vehicles.  A  statistical  model  was  made  of  the  laboratory  results  that  gave  the 
probability  of  detection  versus  range  according  to  a  logistic  curve  fit  and  good  statistical  fits  were 
obtained  from  the  data.  A  fuzzy  logic  model  was  also  made  from  the  data  that  had  0.9  correlation  to 
data  not  used  in  the  training  set. 
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